Computational imaging techniques, sensitive photon 
detectors and considerable imagination are creating 
techniques for 3-D imaging of objects around 
corners and behind walls. 
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ight travels in straight lines. This simple 

observation was the basis, in the 17" cen- 

tury, for Isaac Newton's corpuscular theory 

of light. In the absence of any external force, 

particles travel in straight lines, so a “particle” 
theory of light, such as Newton’s, can capture much of 
what we see in our everyday experience. 

We now know, of course, that things are slightly 
more complicated than that. Thomas Young and 
James Clerk Maxwell showed that light propagates 
as a wave. And then, somewhat ironically, experi- 
ments by Heinrich Hertz that proved Maxwell’s 
equations to be correct also provided the first hints 
of the photoelectric effect. The explanation for that 
effect—that light-matter interaction is quantized— 
earned Albert Einstein his Nobel Prize in Physics, 
and showed that light, while wave-like, is also made 


up of individual, particle-like photons. 


Light in flight 
Two snapshots of a 500-ps, 532-nm laser pulse captured in 
flight as it propagates in free space between two mirrors. 
The white square indicates the area imaged by the SPAD 


camera. The background scene is taken with a standard 
DSLR camera. 
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Hence the fundamental observation and starting 
point for Newton’s reasoning remains true: photons 
do not propagate in curved lines. All of our textbook 
ray-tracing discussions about how images form would 
look very different in a world in which light could bend 
and curve. Granted, astrophysicists enjoy a glimpse of 
such a world through the phenomenon of gravitational 
lensing, in which a gravitational field can effectively 
bend the propagation direction of light as seen by a 
distant observer—thereby allowing a telescope on 
Earth to observe stars and objects otherwise hidden 
(for example, by a distant galaxy) from the direct, 
Euclidean line of sight. Such lensing, however, requires 
huge gravitational forces and propagation over astro- 
nomical distances. 

Here on Earth, we are still bound by the geometri- 
cal description of light rays propagating in straight 
lines. Yet researchers have worked intensely in recent 
years on methods that allow, within the constraint 
of straight light rays, the imaging of objects behind 
obstacles and around corners, developing some 
remarkable imaging technologies and applications 
ina range of fields. 


Capturing light in flight 

This story starts with the first attempts to capture ultra- 
short bursts or pulses of light as these propagate through 
an experiment or through a scene—developments that 
paved the way for the sensitive light detection used in 
non-line-of-sight (NLOS) imaging. 

The first techniques aimed at time-gated imaging 
of a scene were based on mechanical, electronic and 
stroboscope approaches that can provide microsecond 
temporal resolution. Such resolution is sufficient, for 
example, to freeze the motion of a supersonic bullet, as 
pioneered in the remarkable stroboscopic photographs 
by OSA Fellow Harold Edgerton of the Massachusetts 
Institute of Technology, USA, beginning in the late 
1930s. But this is far from the million-fold increase 
in temporal resolution required to freeze the motion 
of a light pulse. 

The first attempts to do that date back to pioneering 
work at Bell Labs in the late 1960s, where laser pulses 
were used to optically control a nonlinear material that 
could act as an ultrafast gate placed in front of a cam- 
era. Similar pump-probe techniques are now widely 
used in most ultrafast-optics laboratories around the 
world, and have become a cornerstone of chemical and 
condensed-matter spectroscopy. 


Researchers have worked intensely in recent years on 
methods that allow the imaging of objects behind obstacles 


and around corners. 


OSA Fellow Nils Abramson, a holography pioneer 
working at the Royal Institute of Technology, Sweden, 
in the 1970s, approached the problem from a concep- 
tually different viewpoint. No longer interested in 
observing ultrafast laser pulse dynamics, Abramson 
focused on capturing light as it propagates through a 
natural scene—for example, light bouncing off a mir- 
ror or propagating in a room. He based his approach 
on an elegant variation of holography, in which the 
light pulse reflected from the natural scene is made 
to interfere with a reference pulse on a photographic 
plate (as in standard holography), with the reference 
pulse propagating at an angle. 

The reference pulse therefore intersects the pho- 
tographic plate in different positions at different 
times. The result is a hologram that, when viewed 
at different positions on the plate, reproduces the 
scene at different times, with picosecond resolution. 
Abramson’s holograms from the early 1970s are— 
together with the high-speed movies created nearly 
100 years earlier by Eadweard Muybridge—among 
the most remarkable photographic-film-based scien- 
tific achievements of all time. 


From ultrafast imaging to 

imaging behind walls 

Moving forward nearly 40 years, Ramesh Raskar 
delivered a TED talk—based on a paper pre- 
sented at the 2011 meeting of the Association of 
Computing Machinery’s Special Interest Group on 
Computer Graphics and Interactive Technologies 
(SIGGRAPH) -—that captured the imagination of 
many (including the author of this feature). The 
paper, “Slow art with a trillion frames per second 
camera,” explicitly alluded to Edgerton’s work 
combining art with technology. In his talk, Raskar 
presented a digital video of a light pulse propagat- 
ing inside a soda bottle filled with water, showing 
exquisite details and even relativistic distortion 
effects of the refracted light. The image had been 
captured using a streak camera—a camera for imag- 
ing ultrafast pulses that relies on a photocathode 


Sample applications for 
non-line-of-sight imaging 
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SELF-DRIVINGIGARS 


The ability to see around corners could find 
applications in future self-driving cars, allowing 
cars to have awareness of approaching vehicles 
or people well before they are directly visible. 
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RESCUE OR MILITARY MISSIONS 


Rescue or military missions will be able to rely on 
the ability to detect the presence of people inside a 
room without actually needing to enter it. 


NASA 


SPACE MISSIONS 


NASA even conducted a study (PERISCOPE) to 
determine whether the technology can remotely 
provide information about the inner structure of lava 
tubes on the moon or remote planets from orbiting 
satellites. These could one day be candidate sites for 
future human settlements. 
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How to image behind a wall 


screen 


hidden 
object 


pulsed 
laser 


1. Scattered light from laser pulses on a screen 
propagates behind the wall into the hidden scene 


\ 


camera/detector 


2. Hidden objects scatter light back toward the screen; 
the detector then measures the arrival time, t, of the 
return signal for a given position on the screen 


3. Time-of-flight information is used to reconstruct 
a series of ellipsoids that all overlap at the 
position of the hidden object 
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to turn photons into electrons. These electrons 
pass between a pair of electrodes and can thus be 
swept across a screen, mapping different temporal 
sections onto different spatial positions. 

Our group, then at Heriot-Watt University, 
U.K., and in collaboration with the University of 
Edinburgh, U.K., subsequently developed this 
further with the use of single-photon avalanche 
photodiode (SPAD) array cameras. These cam- 
eras allowed us to capture the flight of picosecond 
and femtosecond pulses of light in real time (for 
example, with capture times of the order of one 
second), propagating in air (that is, without the 
need for additional scattering agents, as had been 
required in all previous studies). The work, which 
featured videos of light propagating in free space, 
effectively introduced SPAD detectors to the high- 
speed-imaging scene. The SPAD detectors also 
offered a promising route toward real-time 3-D 
imaging of objects that are hidden from view—a 
problem that, as will be seen, hinges critically on 
dealing with a severely limited photon budget. 

Returning to the problem in 2012, Raskar’s 
group, in a paper led by Andreas Velten, showed 
that it is possible to reconstruct a full 3-D image 
of an object hidden behind a wall. In these experi- 
ments, the team used a streak camera to image faint 
“echoes” of light reflected from the hidden scene. 
In this setup, laser pulses directed at the screen 
scatter off of it, diffusing light behind the wall 
and illuminating the hidden object; this object, 
in turn, reflects the light back toward the screen 
on to an area that is visible from the general loca- 
tion of the laser and can thus be picked up by the 
streak camera. 

If the hidden object is a small sphere, for example, 
the returned scattered signal seen by the camera 
can be visualized as an expanding spherical wave, 
as if it were originating from a point-like spheri- 
cal source placed behind the wall. Simplifying the 
concept still further, the shape of the expanding 
spherical light intersecting the wall and imaged by 
the camera—for example, the direction in which 
the sphere propagates—will give direct informa- 
tion of the spatial coordinates of the hidden object. 
Moreover, the time of flight measured for the light 
pulse to leave the laser and finally return back to 
the field of view of the camera provides a measure- 
ment of the distance of the object behind the wall. 


A complicated object can be thought of simply as an ensemble 
of voxels—that is, small individual scatterers, each providing its 
own signal that together will add up in the final camera image. 


Building up complexity 

More complicated objects than spheres can be thought 
of simply as an ensemble of voxels—that is, small 
individual scatterers, each providing its own signal 
that together will add up in the final camera image. 
Combining the information of all these signals, it is 
possible to build a full 3-D image of the hidden object. 

A common approach to reconstructing the loca- 
tion of a voxel that originates a signal from behind a 
wall is back-projection. In essence, the time-of-flight 
(t) measured by the camera is related to the distance 
of the voxel/object from the laser spot on the first wall 
(r,) and the distance of the voxel/object from the obser- 
vation point on the wall (r,): r, + r, = tc, where c is the 
speed of light. This equation describes an ellipsoid, 
with the laser spot and observation point as the foci 
and the overall size determined by the time-of-flight 
distance of the object, tc. 

This single ellipsoid provides an estimate of all 
possible points that could have originated the signal 
observed on the screen at time t. By then either mov- 
ing the laser spot to different positions or, equivalently, 
repeating the measurement for multiple observation 
spots, one can reconstruct a series of ellipsoids, all of 
which overlap at the common voxel/object that origi- 
nated the return signals. How all these ellipsoids are 
used largely depends on the choice of reconstruction 
algorithm or on the kind of information being searched 
for. Full 3-D reconstruction of a hidden object can be 
obtained by adding all of the ellipsoids together and then 
applying filters to threshold and “sharpen” the result. 

Alternatively, if one is interested simply in detect- 
ing the presence and motion of a hidden object, then a 
multiplication of the ellipsoids will provide an estimate 
of object location with minimal computational effort. 
This approach was used by our group to provide the 
first real-time measurements of a hidden object, first 
in the lab and then of humans located around corners 
more than 50 m away from the detection system. 

Alongside the relaxed computational require- 
ments, a number of other factors make the location 
and tracking of an object easier than full 3-D imaging. 


Locating an object is essentially a triangulation prob- 
lem; hence, only three detection points are required. 
If one can make an assumption about the target 
object—for example, that it’s a human or a car that 
will therefore have a fixed height—only two detec- 
tion points will serve to locate the object’s position 
in the horizontal plane. This removes the need for 
any form of scanning; measurements can be updated 
at 1 Hz or faster rates. 

Recent developments by Matthew O'Toole and 
colleagues at Stanford University, Calif., USA, that 
combined a simplified retrieval approach with a con- 
focal lidar-like transceiver (that is, one that co-aligns 
the laser beam and detection) showed full 3-D recon- 
struction of hidden objects in second time scales. This 
could introduce a paradigm change compared with 
the several hours of data acquisition and processing 
of the original streak-camera approach of Velten and 
colleagues, published only six years earlier. 


Photon budget: The SPAD advantage 


All of these techniques ultimately face the problem 
of the total photon budget. The return signal to the 
detector results from two consecutive scattering 
events—thatis, the signal decays as 1/r,?r,”. This leads 
to a severe photon loss—one that scales so badly that 
simply turning up the laser power offers little help. 

For example, a typical long-range experiment 
would use a femtosecond- or picosecond-pulsed, MHz- 
repetition-rate laser with an average power of 0.5 W, 
corresponding to roughly 10!8 photons per second at 
the laser source. Only 10° to 10* photons per second 
are detected in the return signal, which implies sig- 
nificantly less than one return photon per laser pulse, 
and a loss of order of 10". To significantly increase the 
return photons so that, for example, photon-counting 
equipment would no longer be necessary, the laser 
power would need to be increased by three to four 
orders of magnitude—a proposition that creates its 
own problems. 

This highlights the key role played by single- 
photon detection capability in moving these imaging 
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Tracking moving humans 
around corners 


A telescope (subsequently replaced with 
standard 1-inch optics) and a 1530-nm 0.5-W 
pulsed laser source, combined with InGaAs 
SPAD detectors, are used to track the motion 
of a person around a corner more than 50 
meters away from the observer. 


Reconstruction of person’s movements 
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Illustration by Ella Marushchenko; insets: adapted from S. Chan et al., Opt. Express 25, 10109 (2017) 


applications forward. Fortunately, SPAD technology is 
mature and is available also in arrayed detectors with 
sub-100-ps temporal resolutions. As can be intuitively 
seen from the ellipsoid equation discussed earlier, 
the detector’s temporal resolution will determine the 
retrieved 3-D image’s spatial resolution: 100-ps tem- 
poral resolution can provide roughly 1.5-cm distance 
(and therefore shape) resolution. That is sufficient for 
most large area objects, and should make it possible 
to distinguish between main object categories—for 
example, between a human being, a car and a table. 
In recent work, our group took a slightly differ- 
ent approach to determining the shape of a hidden 
object. If the object shape is encoded in the temporal 
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distribution of the return photons, then it should be 
possible to retrieve at least part of this “shape” informa- 
tion from the time-signal recorded with a single-pixel 
detector. By training an artificial neural network (ANN) 
to recognize the differences between three different 
individuals hidden behind a wall (using a 1024-pixel 
SPAD array to parallelize the data acquisition), it was 
possible to acquire new data with one of the individu- 
als and, from the temporal histogram of just one pixel, 
to correctly classify both the position and the identity 
of the hidden person. 

As this discussion shows, there are inherent trade- 
offs involved between computational power and the 
complexity of the final output. The complexity can 


Computer scientists, laser physicists and photon detection 
experts are coming together in the search for a new generation 
of cameras that can look behind the scenes. 


range from a full 3-D scene reconstruction, requiring 
expensive hardware and significant computational 
effort; to simpler object tracking, cheaper in both hard- 
ware and computational effort; to ANNs for object 
identification, which are cheap in optical hardware 
but expensive in computational power (although 
that computation is front-loaded in the ANN train- 
ing session). 


The future around the corner 


This feature has focused on the “mainstream” 
approach to imaging behind walls—that is, on tech- 
niques that rely on active illumination of the scene 
with a pulsed laser and the detection of return sig- 
nals. Other proposed approaches could find use 
in specific scenarios, and should be considered in 
future developments. For example, continuous- 
wave lasers rather than pulsed lasers might be used, 
where the object shape is reconstructed from distor- 
tions induced by the object in the reflected “glare” 
observed on the screen, as proposed by Jonathan 
Klein and colleagues. 

The possibility of using speckle patterns and the 
speckle memory effect in reflection from a scattering 
surface has also emerged, and recent work from the lab 
of OSA Fellow Aristide Dogariu of CREOL, University 
of Central Florida, USA, has suggested looking at signa- 
tures in spatial coherence of the signal that is received 
from behind the wall. This latter approach raises the 
potential for fully passive sensing thatno longer requires 
any form of illumination and that reconstructs the 3-D 
scene from shadows and statistical properties of the 
photons that make the journey from behind the wall 
to the detector. 

Computational imaging techniques and single- 
photon sensors are rapidly progressing, and the 
fusion of these two research areas is producing new 
imaging capabilities—not merely enhancements of 
conventional imaging, but paradigm-changing ideas 
in which seemingly impossible feats are performed. 
There are currently high expectations in the field, 
with hopes that it will soon produce cameras that 


can see through opaque media, inside the body and 
(why not?) also behind walls. 

The major challenges to be addressed involve the 
difficulty in achieving real-time 3-D imaging with a 
device that is safe for the eyes, portable and easy to 
use. These are largely engineering problems—although 
the nature of these is such that systems may need to 
be completely redesigned or even based on different 
principles from those currently in use. These chal- 
lenges are bringing together computer scientists, laser 
physicists and photon detection experts in the search 
for anew generation of cameras that can look behind 
the scenes. 


OSA Fellow Daniele Faccio (Daniele.Facciofaglasgow.ac.uk) 
is with the School of Physics and Astronomy, University 
of Glasgow, U.K. 
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